Gesture-enabled keyboard and associated apparatus and computer-readable storage medium

ABSTRACT

A keyboard is provided that includes an arrangement of keys, circuitry and a first processor configured to generate a signal in response to a keystroke in which a respective key of the arrangement of keys is depressed, and determine how to interpret the keystroke. The keyboard also includes a plurality of proximity sensors and a second processor configured to measure the proximity of an object to the respective proximity sensors, determine a position and/or movement of the object relative to the proximity sensors based on the respective signal, and produce a gesture signal representative of the determined position and/or movement. This gesture signal is interpretable into one or more commands or instructions for directing performance of one or more operations of an apparatus or software operating on the apparatus. A related apparatus and computer-readable storage medium are also provided.

FIELD OF THE INVENTION

The present invention generally relates to a user interface and methodsfor interacting with a computer system, and more particularly, to agesture-enabled keyboard and associated apparatus and computer-readablestorage medium.

BACKGROUND OF THE INVENTION

In the field of medical imaging, prior to the digitization of medicalimaging, medical-imaging users (e.g., Radiologists) would analyzephysical film printed images in light boxes, and use physical devicessuch as magnifying glasses, rulers, grease pencils, and their hands tomanipulate the physical printed medical images in order to interpret anddiagnose the images. With the digitization of medical imaging, thephysical film became a digital image, displayable on a computer monitor.A medical-imaging system became a computer application or collection ofcomputer applications, which require a computer or computers to operate.At present, medical-imaging systems are interacted with through a mouseand keyboard. Commands to the medical-imaging system are typicallyinvoked through mouse and/or keyboard interactions.

For image-intensive computing with rich graphical user interfaces, themouse is showing its age. The mouse constrains the interaction to asingle x, y point on the display with buttons to make selections andinitiate modes of operation, such as click and drag. Most moderncomputer mice also have a special purpose scroll mechanism, often awheel. Much of the human hand and finger capabilities and dexterity arenot utilized with a mouse, and a mouse is limited to only one hand.Using the mouse for long periods of time tends to cause fatigue andrepetitive stress symptoms due to the need to grasp the device andrepeatedly perform small stressing motions of the fingers to position,click and scroll.

For alphanumeric text entry, and initiating commands, keyboards haveremained much the same for many decades and it has been difficult tofind alternatives that perform better at text entry, for most users.When used as an input device for medical-imaging systems, some form ofkeyboard may remain in use for text entry for a long time to come.

Another common purpose for which keyboards are used with medical-imagingsystems is for shortcut commands to activate discrete operations alsoavailable with the mouse but at the cost of pointer navigation time andadditional mouse clicking. Most medical imaging professionals whocommonly use picture archiving and communication system (PACS)workstations learn a number of these keyboard shortcuts to perform theirwork faster, and are willing to spend the effort to remember what keysmap to what commands. For some interactive operations, shortcuts mayalso involve mouse modes of operation, often in conjunction with thekeyboard. For example, a medical-imaging system zoom-image functioncould be mapped to the combination of holding down the Ctrl key andmoving the mouse forward and back or rolling the scroll wheel. Thesecommon interactive operations that involve the mouse are ones that oftentake the most time to execute and cause the majority of repetitivestrain injury (RSI) and fatigue issues.

SUMMARY OF THE INVENTION

The continued presence and usage of a physical keyboard with PACSworkstations mean it can be used as a base for augmentation withadditional sensors to enable fast interactive operations and withoutcausing any additional clutter or hardware complexity for users. Theuser can maintain their hand position at the keyboard for longerperiods, use the mouse less and perform common interactive operationsquickly with low stress and fatigue. Exemplary embodiments of thepresent invention therefore provide an improved apparatus and method formore intuitively and efficiently interacting with a computer system,such as a medical-imaging system. According to one aspect of exemplaryembodiments of the present invention, a keyboard is provided thatincludes an arrangement of keys, circuitry and a first processor. Thecircuitry forms a grid of circuits underneath respective keys of thearrangement of keys. In this regard, a circuit of the grid of circuitsis configured to generate a signal in response to a keystroke in which arespective key of the arrangement of keys is depressed. The processor,then, is configured to receive the signal from the circuit of the gridof circuits, and compare a location of the respective circuit to acharacter map to determine how to interpret the keystroke.

The keyboard of this aspect further includes a plurality of proximitysensors and a second processor. The proximity sensors form a grid ofsensors underneath the keys; and are configured to measure the proximityof an object (hand or finger(s) of a hand of a user) to the respectiveproximity sensors, and produce a signal representative of the measuredproximity. The second processor, then, is configured to receive thesignal representative of the measured proximity, determine a positionand/or movement of the object relative to the proximity sensors based onthe respective signal, and produce a gesture signal representative ofthe determined position and/or movement. This gesture signal isinterpretable into one or more commands or instructions for directingperformance of one or more operations of an apparatus or softwareoperating on the apparatus.

The plurality of proximity sensors may include a first plurality ofproximity sensors configured to project respective electric fieldsthrough the keys and upward from a top surface of the keyboard.Additionally, the plurality of proximity sensors may also include asecond plurality of proximity sensors configured to project respectiveelectric fields outward from sides of the keyboard. The second pluralityof proximity sensors may additionally or alternatively include proximitysensors configured to project respective electric fields outward from aback of the keyboard. And even further, the second plurality ofproximity sensors may additionally or alternatively include proximitysensors underneath a rest pad extending from a front of the keyboard,where these sensors are configured to project respective electric fieldsupward and outward from the rest pad.

According to another aspect of exemplary embodiments of the presentinvention, an apparatus is provided that includes a processor configuredto at least perform or cause the apparatus to at least perform a numberof operations. The operations include receiving a signal from a keyboardthat, similar to the keyboard described above, includes an arrangementof keys, plurality of proximity sensors and a second processor. Theoperations also include determining one or more commands or instructionsfor directing performance of one or more operations of the apparatus orsoftware operating on the apparatus, where the command(s) orinstruction(s) are determined as a function of the signal received fromthe keyboard. In this regard, the command(s) or instruction(s) mayinclude command(s) or instruction(s) that effectuate a change in agraphical output presented by a display during operation of the softwareoperating on the apparatus.

A signal may be received from the keyboard in at least a first instanceand a second instance, where the signal in the first instance includes atrigger signal, and the signal in the second instance includes a gesturesignal. In the first instance, the trigger signal is received inresponse to a user performing a predefined trigger gesture movement withrespect to the keyboard. Also in the first instance, the processor isconfigured to perform or cause the apparatus to perform identificationof the signal received from the keyboard as a trigger signal to therebyturn on gesture recognition and interpretation. And in these instances,the command(s) or instruction(s) may be determined in the secondinstance after interpreting the trigger signal in the first instance,where the processor is otherwise configured to ignore the gesturesignal.

More particularly, for example, in the first instance, identifying asignal as a trigger signal may include identifying the signal as beingrepresentative of a determined position and/or movement of the user'shand/finger(s) corresponding to the predefined trigger gesture movementof the hand/finger(s). The predefined trigger gesture movement mayinclude sliding the finger(s) from a corner of a back edge of thekeyboard along the back edge so that the finger(s) cover a distancewithin a predefined range of distances along the back edge. As anotherexample, the predefined trigger gesture movement may include resting thehand or one or more fingers over a side edge of the keyboard, with thehand or one or more fingers covering all or only half of the side edgeof the keyboard. And as another example, the predefined trigger gesturemovement may include resting the hand or one or more fingers over theside edge of the keyboard, with a thumb of the hand resting over a frontedge of the keyboard.

According to other aspects of exemplary embodiments of the presentinvention, a computer-readable storage medium is provided. Exemplaryembodiments of the present invention therefore provide a keyboardincluding proximity sensors, as well as an apparatus andcomputer-readable storage medium for directing operation of an apparatusor software operating on the apparatus via keyboard including proximitysensors. As indicated above, and explained below, exemplary embodimentsof the present invention may solve problems identified by priortechniques and provide additional advantages.

BRIEF DESCRIPTION OF THE DRAWINGS

Having thus described the invention in general terms, reference will nowbe made to the accompanying drawings, which are not necessarily drawn toscale, and wherein:

FIG. 1 is a schematic block diagram of an apparatus configured tooperate in accordance with embodiments of the present invention;

FIG. 2 is a schematic top view of a gesture-enabled keyboard, accordingto exemplary embodiments of the present invention;

FIG. 3 is a schematic top view of proximity sensors underneath a topsurface of the gesture-enabled keyboard of FIG. 2, according toexemplary embodiments of the present invention;

FIG. 4 is a schematic perspective view of the gesture-enabled keyboardof FIG. 2 showing a number of the proximity sensors underneath the topsurface, according to exemplary embodiments of the present invention;

FIG. 5 is a flowchart illustrating various steps in a method ofreceiving and interpreting gesture signals into gesture commands orother instructions, according to exemplary embodiments of the presentinvention;

FIG. 6 is a schematic top view of a trigger mechanism, according toexemplary embodiments of the present invention; and

FIGS. 7-14 are schematic views of various example gesture movements andtrigger mechanisms that may be employed according to exemplaryembodiments of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention now will be described more fully hereinafter withreference to the accompanying drawings, in which preferred embodimentsof the invention are shown. This invention may, however, be embodied inmany different forms and should not be construed as limited to theembodiments set forth herein; rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the scope of the invention to those skilled in the art. Forexample, references may be made herein to directions and orientationsincluding vertical, horizontal, diagonal, right, left, front, back andside; it should be understood, however, that any direction andorientation references are simply examples and that any particulardirection or orientation may depend on the particular object, and/or theorientation of the particular object, with which the direction ororientation reference is made. Like numbers refer to like elementsthroughout.

Referring to FIG. 1, a block diagram of one type of apparatus configuredaccording to exemplary embodiments of the present invention is provided(“exemplary” as used herein referring to “serving as an example,instance or illustration”). The apparatus and method of exemplaryembodiments of the present invention will be primarily described inconjunction with medical-imaging applications, such as in the context ofa PACS workstation. It should be understood, however, that the methodand apparatus of embodiments of the present invention can be utilized inconjunction with a variety of other applications, both in the medicalindustry and outside of the medical industry. Further, the apparatus ofexemplary embodiments of the present invention includes various meansfor performing one or more functions in accordance with exemplaryembodiments of the present invention, including those more particularlyshown and described herein. It should be understood, however, that oneor more of the entities may include alternative means for performing oneor more like functions, without departing from the spirit and scope ofthe present invention.

Generally, the apparatus of exemplary embodiments of the presentinvention may comprise, include or be embodied in one or more fixedelectronic devices, such as one or more of a laptop computer, desktopcomputer, workstation computer, server computer or the like. In a moreparticular example, the apparatus may comprise, include or be embodiedin a picture archiving and communication system (PACS) or othermedical-imaging system workstation. Additionally or alternatively, theapparatus may comprise, include or be embodied in one or more portableelectronic devices, such as one or more of a tablet computer, mobiletelephone, portable digital assistant (PDA) or the like.

As shown in FIG. 1, the apparatus 10 of one exemplary embodiment of thepresent invention may include a processor 12 connected to a memory 14.The memory can comprise volatile and/or non-volatile memory, andtypically stores content, data or the like. In this regard, the memorymay store content transmitted from, and/or received by, the apparatus.The processor and/or memory may be embodied in any of a number ofdifferent manners including, for example, one or more of any of thefollowing: microprocessors, processors with or without accompanyingdigital signal processor(s), special-purpose integrated circuits,field-programmable gate arrays (FPGAs), controllers,application-specific integrated circuits (ASICs), computers or the like.

The memory 14 may also store one or more software applications 16,instructions or the like for the processor to perform steps associatedwith operation of the entity in accordance with exemplary embodiments ofthe present invention (although any one or more of these steps may beimplemented in hardware alone or in any combination with software and/orfirmware). This software may include, for example, agesture-interpretation engine configured to receive gesture signal(s)and interpret those signal(s) to direct performance of one or morefunctions of the apparatus. In addition, the software may includesoftware applications (e.g., medical-imaging software, Internet browser,etc.), one or more operations of which may be directed by thegesture-interpretation engine (and, hence, the user of the apparatus viainteraction with a gesture-enabled keyboard).

In addition to the memory 14, the processor 12 may also be connected toat least one interface or other means for displaying, transmittingand/or receiving data, content or the like. In this regard, theinterface(s) may include at least one communication interface 18 orother means for transmitting and/or receiving data, content or the like,such as to and/or from other device(s) and/or network(s) coupled to theapparatus. In addition to the communication interface(s), theinterface(s) may also include at least one user interface that mayinclude one or more wireline and/or wireless (e.g., Bluetooth) earphonesand/or speakers, one or more displays 20, and/or a user input interface22. The user input interface, in turn, may comprise any of a number ofwireline and/or wireless devices allowing the entity to receive datafrom a user, such as a microphone, an image or video capture device, akeyboard or keypad, a joystick, or other input device.

As shown in FIG. 2, according to a more particular exemplary embodiment,the user input interface 22 may include a gesture-enabled keyboard 24.Similar to a conventional keyboard, the gesture-enabled keyboardincludes an arrangement of buttons or keys 26 (key generally referringto a button or key), and may include a processor (and memory) andcircuitry configured to interpret keystrokes on the keyboard. Theprocessor and memory may be embodied in any of a number of differentmanners, including any of those described above. The keys of thekeyboard may be depressed by a user to produce letters, numbers ofsymbols for input into a software application operating on theapparatus, and/or that may be depressed to effectuate functions of theapparatus or a software application operating on the apparatus. Althoughdescribed herein as a keyboard, the gesture-enabled keyboard ofexemplary embodiments of the present invention may additionally oralternatively include a keypad or other similar arrangement of keys orbuttons.

Generally, the circuitry of the gesture-enabled keyboard 24 forms a gridof circuits underneath the keys 26 to form a key matrix. The circuitsare configured to generate signals in response to the user depressingkeys of the keyboard. For example, the circuits may be broken underneaththe keys such that, when a user depresses one or more keys, thecircuit(s) underneath the respective key(s) is completed; therebygenerating signal(s) (e.g., current(s)). The processor receives thesignal(s) from the respective circuit(s), and compares the location(s)of the circuit(s) to a character map (lookup table) in its memory todetermine how to interpret the keystroke(s). As will be appreciated,although the keyboard may include mechanical-type switches that operateas above, the keyboard may alternatively employ non-mechanical switchingtechniques based on resistive or capacitive techniques.

As shown in FIGS. 3 and 4, in accordance with exemplary embodiments ofthe present invention, the circuitry of the gesture-enabled keyboard 24further includes a plurality of proximity sensors 28 and associatedcircuitry. The associated circuitry may include a processor and/ormemory, which may be embodied in any of a number of different manners,including any of those described above. An example of a suitableproximity sensor includes capacitive proximity sensors such as thoseimplementing projected capacitance techniques, although other similarproximity sensors may be employed. And suitable sensors and associatedcircuitry includes, for example, the GestIC® technology developed byIDENT Technology AG of Germany.

The proximity sensors 28 may be situated in any of a number of differentrelationships relative to keys 26 of the keyboard 24, and may compriseany of a number of different types of known sensors configured tomeasure proximity of an object to the respective sensors and producecorresponding signals as a function of the measured proximity. Forexample, the proximity sensors may overlay or lie underneath the keys ofthe keyboard, or the proximity sensors may be integrated within therespective keys. Additionally, proximity sensors may overlay or lieunderneath outer edges and/or a rest pad 30 of the keyboard.

In one example embodiment, the proximity sensors 28 may include a gridof sensors underneath keys 26 of the keyboard 24 and configured suchthat their electric fields (e-fields or sensing fields) are directedthrough the keys and upward from a top surface the keyboard. In thisexample embodiment, the proximity sensors also include an additionalnumber of sensors along side and back edges of the keyboard andconfigured such that their e-fields are directed outward from the sidesand back of the keyboard, and may further include a number of sensorsunderneath the rest pad of the keyboard and configured such that theire-fields are directed upward and outward from the rest pad.

The proximity sensors 28 of the gesture-enabled keyboard 24 areconfigured to detect an object such as a user's hand proximate therespective sensors, and produce signals representative of the proximityof detected object to the respective sensors. The processor of thecircuitry associated with the proximity sensors may be configured toreceive and process the signals to determine the position and/ormovement of the detected object relative to the proximity sensors (andhence, the keyboard), and to produce signal(s) representative of thedetermined position and/or movement, and possibly other relatedinformation such as velocity. These signal(s) (referred to herein as“gesture signals”), then, may be provided in addition to and independentof the interpretation of a keystroke determined by the processor of thekeyboard.

In accordance with exemplary embodiments of the present invention, thegesture-enabled keyboard 24 may be configured to provide gesture signalsand any keystroke interpretations to the device 10. Referring to FIG. 5,the gesture-interpretation engine (software 16) of the device may beconfigured to receive and interpret the respective gesture signals intogesture commands or other instructions for directing performance of oneor more operations of the device, or more particularly in variousinstances, operations of other software operating on the device. At anyinstant in time, the sensors 28 and gesture-interpretation engine may becapable of detecting and interpreting gesture signals indicative of asingle sensed point (single-sense) or multiple simultaneous sensedpoints (multi-sense). In various instances, execution of operations ofthe device may effectuate a change in a graphical output presented bythe display 12 during operation of the other software.

The gesture-interpretation engine (software 16) may be configured tointerpret the gesture signals and direct performance of one or moreoperations of the device 10 as a function of the respective signals. Thegesture-interpretation engine may be configured to distinguish betweenmovements of the user intended as gesture movements and other movementsnot intended as gesture movements, such as those made during interactionwith keys 26 of the keyboard 24. For example, the gesture-interpretationengine may include a trigger mechanism to “turn on” gesture recognitionand interpretation—the device otherwise ignoring gesture signals fromthe keyboard. The device may respond to the user employing the triggermechanism before or concurrent with all gesture movements. Or in variousinstances, the device may respond to the trigger mechanism for variousgesture movements that resemble other movements not intended as gesturemovements, and may respond to other gesture movements without thetrigger mechanism. The device and/or keyboard in these instances or moregenerally may provide feedback, such as aural and/or visual feedback(e.g., on the display 20 and/or by a visual indicator such as alight-emitting diode (LED) or the like), indicating that the keyboard isready to receive—or is receiving—a gesture movement.

Examples of gesture movements that may more particularly benefit fromuse of a trigger mechanism include the circling of finger(s), wiping orstroking finger(s) front to back, waving finger(s) and/or hand side toside, and moving fingers and/or hand up and down towards and away fromthe top surface of keyboard. In another example, a user may move theirfingers in multi-finger combinations, such as to effect a typicalzoom/pan/rotate (e.g., scale/translate/rotate) finger and thumbcombination on one hand where one moves their fingers toward or awayfrom one another to zoom, move both over the surface to pan, or rotatethe vector between the two to rotate. And in another example, a user may“flick” their fingers in a manner interpreted as a quick motion in onedirection such as left to right, or front to back, or vise versa.

Examples of gesture movements that may not particularly benefit from useof a trigger mechanism (but for which a trigger mechanism may berequired or otherwise implemented) include sliding finger(s) side toside along an edge (e.g., front edge) of the keyboard, and slidingfinger(s) back and forward along an edge (e.g., side edge) of thekeyboard (e.g., opposite side edge to the one configured for turning ongesture recognition and interpretation. And in another example, the usermay slide their finger(s) side to side along the top surface of thekeyboard, such as along the back top surface behind a row of functionkeys over sensors 28 whose e-fields are directed upward from thekeyboard.

The trigger mechanism according to one example embodiment may includethe user performing a trigger gesture movement, such as by resting theuser's hand or finger(s) of the user's hand over a side edge of thekeyboard 24 and possibly also resting a thumb or finger(s) over a frontor back edge near the corners of the keyboard, as shown in FIG. 6. Whenthe user rests their hand, thumb and/or finger(s) in this manner, thesensor(s) 28 along the respective edge(s) may produce appropriatesignals processed by the processor of the sensor's associated circuitryinto a trigger signal, which the device may identify as a trigger signalto turn on gesture recognition and interpretation. In such instances,the user may place one hand to indicate “turn on” the keyboard toreceive a gesture movement—and hence direct the keyboard to produce atrigger signal; and the user may move their other hand above the keys(and the proximity sensors 28) to effect a gesture movement. Othertrigger mechanisms may additionally or alternatively be employed, someof which may be particular for “turning on” particular gesturemovements. Examples of other trigger mechanisms are described below.

The gesture-interpretation engine (software 16) of the device 10 may beconfigured to process (interpret and direct device operations as afunction of) the gesture signals as the engine receives the respectiveinformation. Alternatively, however, the engine may be configured tostore the information for later processing or analysis by the engine,device or other software of the device. Additionally or alternatively,the device may be configured to transmit the information over a networkto one or more other devices, where the information may undergoprocessing and/or analysis.

The gesture-interpretation engine may be configured to process thegesture signals and any other of the information in any of a number ofdifferent manners. As indicated above, for example, the engine may beconfigured to process the gesture signals to identify one or moregesture movements of the user and direct performance of one or moreoperations of the device 10, or more particularly in various instances,operations of other software operating on the device, as a function ofthe respective gesture movement(s). In this regard, the device may storea mapping between a distinct set of gestures movements and respectivefunctions of the apparatus 10 or software application. That is, thegesture-interpretation engine may detect one or more gesture movementsas inputs, and in response thereto, direct respective operations of theapparatus or software application as outputs. The gesture movements maybe implemented in any appropriate sequence, or in various instances,multiple gesture movements may be implemented simultaneously. In thecontext of a PACS workstation, for example, gesture movements may beassociated with imaging operations such as pan within an image or otherdisplay, zoom within an image or other display, change an viewportlayout, scroll through a series of images/displays, adjust animage/display window and level or the like. Other examples of possibleoperations with which gesture movements may be associated include open acurrently-selected study, close a study or the like. Gesture movementsthat may be simultaneously-implementable include those for functionssuch as simultaneous zoom and scroll, zoom and pan, scroll and adjustwindow and level or the like.

More particular examples of various gesture movements and triggermechanisms that may be employed along with the respective movements areprovided below with reference to FIGS. 7-14. It should be understood,however, that the example gesture movements described herein need not betriggered by a trigger mechanism. It should also be understood thatalthough example trigger mechanisms may be described as triggeringrespective, example gesture movements, the trigger mechanisms may beequally employed to trigger other gesture movements, in addition to orin lieu of the gesture movements with which the trigger mechanisms aredescribed as triggering herein.

Referring now to FIGS. 7, 8 and 9, one example gesture movement may beemployed to direct performance of zooming operations within softwareoperating on the device 10. Referring more particularly to FIGS. 7 a, 7b and 7 c, this gesture movement may be triggered by a trigger mechanismin which the user holds one hand over a side edge of the keyboard 24covering only a front half of the side edge. The user may then perform agesture movement to zoom within software operating on the device.

As described herein, “half” of a length of the keyboard may beinterpreted as the portion of the length underneath which half of theproximity sensors 28 along the length are disposed; and similarly, “all”or the “whole” of the length may be interpreted as the portion of thelength underneath which all of the proximity sensors along the lengthare disposed. Thus, for example, for a side edge having four proximitysensors disposed underneath, half of the side edge may be interpreted asthe portion of the side edge underneath which two of the proximitysensors are disposed, and all of the side edge may be interpreted as theportion of the side edge underneath which all four of the proximitysensors are disposed.

In one example embodiment, this zooming gesture movement may include theuser waving finger(s) on their other hand (or generally their otherhand) from one side to the other side over the keyboard to zoom in, orwaving their other hand/finger(s) from the other side to the one sideover the keyboard to zoom out, as shown in FIG. 7 a. In another exampleembodiment, a zooming gesture movement may include the user wiping orstroking finger(s) on their other hand from front to back over thekeyboard to zoom in one direction (e.g., in/out), or wiping or strokingtheir finger(s) from back to front to zoom in the other direction (e.g.,out/in), as shown in FIG. 7 b. In yet another example embodiment, azooming gesture movement may include the user raising finger(s) on theirother hand (or generally their other hand) up away from the top ofkeyboard to zoom in one direction e.g., (in/out), or pushing theirhand/finger(s) down towards the top of keyboard to zoom in the otherdirection (e.g., out/in), as shown in FIG. 7 c.

In performing the zooming gesture movement, or more generally inperforming any of a number of gesture movements, the user may clutch thehand effectuating the gesture movement or between gesture movements. Insuch instances, the user may bring their respective hand or itsfinger(s) off the top area of the keyboard 24 to avoid the sensors 28detecting a constantly reversing direction. Alternatively, thegesture-interpretation engine may be configured to detect and remember(e.g., store an indication of) the starting direction of the gesturemovement so the user need not remove their clutching hand/finger(s) fromthe top area of the keyboard. As another alternative, the user mayremove their hand implementing the trigger mechanism from the keyboardto thereby “turn off” gesture recognition and interpretation.

Referring to FIGS. 8 a, 8 b and 8 c, another example implementing azooming gesture movement includes a trigger mechanism in which the userholds one hand over a side edge of the keyboard 24 covering only a fronthalf of the side edge to enable zoom gesturing in one zoom direction(e.g., in/out). To enable zoom gesturing in the other zoom direction(e.g., out/in), then, the user may slide their same hand back along therespective edge such that their hand rests over all of the side edge.The gesture movements described above with respect to FIGS. 7 a, 7 b and7 c may then be performed without regard to direction of movement of theuser's hand/finger(s) as placement of the user's trigger hand (handperforming the trigger gesture movement) controls the zoom direction(e.g., in/out).

Referring to FIG. 9, in yet a further example implementing a zoominggesture movement, a trigger mechanism may not be needed to turn ongesture recognition and interpretation. In this example, the user mayslide finger(s) of one of their hands side-to-side along the back topsurface of the keyboard 24, such as behind a row of function keys on thekeyboard (shown in FIG. 2, for example, as keys identified by “F” and adigit—e.g., F1, F2, etc.). The user may slide their finger(s) in onedirection to zoom in, and in another direction to zoom out. Also in thisexample, the sensors 28 disposed underneath this area of the keyboardmay be set to project their e-fields only far enough to sense throughthe keyboard case and not far into the air space above (e.g., one or twomillimeters) such that the user has to run their finger(s) along thesurface for detection, thereby avoiding false activations.

FIG. 10 illustrates another example gesture movement may be employed todirect performance of panning operations within software operating onthe device 10. This gesture movement may be triggered by a triggermechanism in which the user holds one hand over a side edge of thekeyboard 24 covering only a front half of the side edge, and lowerstheir thumb of that hand to the front edge of the keyboard, as shown inFIG. 10 a. In another example shown in FIG. 10 b, the panning gesturemovement may be triggered by a trigger mechanism in which the userslides one of their hands to cover the whole of a side edge of thekeyboard (covering multiple sensors 28). In either instance, however,the user may then move finger(s) on their other hand (or generally theirother hand) over the keyboard surface in any X and/or Y direction to panan image or other display of the software.

FIG. 11 illustrates another example gesture movement may be employed todirect performance of rotating operations within software operating onthe device 10 (causing rotation of an image or other display of thesoftware). This gesture movement may be triggered by any of theaforementioned trigger mechanisms. Regardless of the trigger mechanism(or even whether one is required), the user may circle finger(s) ontheir non-trigger hand in one direction over the top surface of thekeyboard 24 to rotate the image/display in one direction, or circlefinger(s) in the other direction to rotate the image/display in theother direction, as shown in FIG. 11 a. In another example, the user mayquickly flick finger(s) on their non-trigger hand over the top surfaceof the keyboard toward the front or back of the keyboard to rotate theimage/display in ninety-degree increments, as shown in FIG. 11 b. Inthis regard, the user flicking their finger(s) front-to-back may rotatethe image/display in one direction, and flicking back-to-front mayrotate the image/display in the other direction.

In various instances, the device 10 may be capable of implementing anycombination of the zooming, panning or rotating operations in responseto an appropriate combination gesture movement, as shown in FIG. 12. Inthese instances, the respective gesture movement may be triggered by atrigger mechanism such as one of the mechanisms described above. Theuser may then implement a multi-sense type gesture movement in which theuser pinches a finger and thumb on one hand together or moves theirfinger and thumb apart to zoom an image/display (in/out), moves boththeir finger and thumb over the top surface of the keyboard 24 to panwithin the image/display, and/or rotates a vector between their fingerand thumb to rotate the image/display.

FIG. 13 illustrates yet another example gesture movement may be employedto direct performance of scrolling operations within software operatingon the device 10 (scrolling through a series of images/displays of thesoftware). Similar to before, this gesture movement may be triggered bya trigger mechanism such as one of the mechanisms described above. Theuser may then move finger(s) on their other hand (or generally theirother hand) front-to-back or back-to-front along a side edge of thekeyboard 24 (if implemented, the side other than that on which thetrigger mechanism is being implemented), as shown in FIG. 13 a. Inanother example, the user may similarly move their finger(s)side-to-side along the front edge of the keyboard, as shown in FIG. 13b. The user sliding their fingers in one direction scrolls through aseries in one direction, and sliding their fingers in the otherdirection scrolls through the series in the other direction. In yetanother example implementing a scrolling gesture movement, the user mayquickly flick finger(s) on their other hand over the keyboard toward oneside or the other to scroll through a series, as shown in FIG. 13 c.Flicking left-to-right may scroll the series in one direction, andflicking right-to-left may scroll the series in the other direction.

FIG. 14 illustrates an additional example gesture movement may beemployed to direct performance of window/level adjustment operationswithin software operating on the device 10 (adjusting an image/displaywindow and level). This gesture movement may be triggered by a triggermechanism in which the user slides finger(s) on one hand from a cornerof the back edge along the back edge of the keyboard 24 so that theirfingers cover a distance within a predefined range of distances (e.g., adistance considered short) along the back edge of the keyboard from therespective corner. The user may then slide their finger(s) side-to-sidein front of or near the front edge of the keyboard, whereby slidingtheir finger(s) in one direction adjusts the window of an image/displayin one direction, and sliding their finger(s) in the other directionadjusts the window in the other direction, as shown in FIG. 14 a.Similarly, the user may slide finger(s) on their other hand (orgenerally their other hand) front-to-back or back-to-front along theother side edge of the keyboard, whereby sliding their finger(s) in onedirection adjusts the level of an image/display in one direction, andsliding their finger(s) in the other direction adjusts the level in theother direction, as shown in FIG. 14 b.

In a further example implementing a window/level adjustment gesturemovement, the user may move finger(s) on their other hand (or generallytheir other hand) over the top surface of the keyboard in any X and/or Ydirection to adjust the window and/or level, as shown in FIG. 14 c. Inthis regard, the user moving their finger(s)/hand in the X direction mayadjust one of the window or level of an image/display, and moving theirfinger(s)/hand in the Y direction may adjust the other of the window orlevel of the image/display. And movement of the user's finger(s)/hand inboth the X and Y directions may simultaneously adjust both the windowand level of the image/display.

As explained herein, the device 10 and the gesture-enabled keyboard 24may each include a processor (e.g., processor 12) and/or circuitry(which may include a processor) configured to perform one or morefunctions. It should be understood that one or more of the functions ofthe processor of the device may instead be performed by the processorand/or circuitry of the gesture-enabled keyboard. Likewise, one or moreof the functions of the processor and/or circuitry of thegesture-enabled keyboard may instead be performed by the processor ofthe device. And even further, one or more of the functions of theprocessor and/or circuitry of either or both of the device orgesture-enabled keyboard may be performed by yet another device, whichmay be directly or indirectly coupled to the device or gesture-enabledkeyboard, such as via one or more network(s).

According to one aspect of the present invention, all or a portion of anapparatus (e.g., device 10, gesture-enabled keyboard 24) generallyoperates under control of a computer program. The computer program forperforming the methods of exemplary embodiments of the present inventionmay include one or more computer-readable program code portions, such asa series of computer instructions, embodied or otherwise stored in anon-transitory computer-readable storage medium, such as thenon-volatile storage medium.

It will be understood that each step of a method according to exemplaryembodiments of the present invention, and combinations of steps in themethod, may be implemented by computer program instructions. Thesecomputer program instructions may be loaded onto a computer or otherprogrammable apparatus to produce a machine, such that the instructionswhich execute on the computer or other programmable apparatus createmeans for implementing the functions specified in the step(s) of themethod. These computer program instructions may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable apparatus to function in a particular manner, such that theinstructions stored in the computer-readable memory produce an articleof manufacture including instruction means which implement steps of themethod. The computer program instructions may also be loaded onto acomputer or other programmable apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions which execute on the computer or other programmableapparatus provide steps for implementing steps of the method.

Accordingly, exemplary embodiments of the present invention supportcombinations of means for performing the specified functions,combinations of steps for performing the specified functions and programinstruction means for performing the specified functions. It will alsobe understood that each step or function, and combinations of steps orfunctions, can be implemented by special purpose hardware-based computersystems which perform the specified functions or steps, or combinationsof special purpose hardware and computer instructions.

Many modifications and other embodiments of the invention will come tomind to one skilled in the art to which this invention pertains havingthe benefit of the teachings presented in the foregoing descriptions andthe associated drawings. It should therefore be understood that theinvention is not to be limited to the specific embodiments disclosed andthat modifications and other embodiments are intended to be includedwithin the scope of the appended claims. Although specific terms areemployed herein, they are used in a generic and descriptive sense onlyand not for purposes of limitation.

What is claimed is:
 1. A keyboard comprising: an arrangement of keys;circuitry forming a grid of circuits underneath respective keys of thearrangement of keys, a circuit of the grid of circuits being configuredto generate a signal in response to a keystroke in which a respectivekey of the arrangement of keys is depressed; a first processorconfigured to receive the signal from the circuit of the grid ofcircuits, and compare a location of the respective circuit to acharacter map to determine how to interpret the keystroke; a pluralityof proximity sensors forming a grid of sensors underneath the keys, theproximity sensors being configured to measure a proximity of an objectto the respective proximity sensors, and produce a signal representativeof the measured proximity; and a second processor configured to receivethe signal representative of the measured proximity, determine at leastone of a position or movement of the object relative to the proximitysensors based on the respective signal, and produce a gesture signalrepresentative of the determined at least one of position or movement,the gesture signal being interpretable into one or more commands orinstructions for directing performance of one or more operations of anapparatus or software operating on the apparatus.
 2. The keyboard ofclaim 1, wherein the plurality of proximity sensors include a firstplurality of proximity sensors configured to project respective electricfields through the keys and upward from a top surface of the keyboard.3. The keyboard of claim 2, wherein the plurality of proximity sensorsalso include a second plurality of proximity sensors configured toproject respective electric fields outward from sides of the keyboard.4. The keyboard of claim 2, wherein the plurality of proximity sensorsalso include a second plurality of proximity sensors configured toproject respective electric fields outward from a back of the keyboard.5. The keyboard of claim 2 further comprising a rest pad extending froma front of the keyboard, wherein the plurality of proximity sensors alsoinclude a second plurality of proximity sensors underneath the rest pad,the second plurality of proximity sensors being configured to projectrespective electric fields upward and outward from the rest pad.
 6. Anapparatus comprising a processor configured to at least perform or causethe apparatus to at least perform the following: receiving a signal froma keyboard, the keyboard comprising: an arrangement of keys; a pluralityof proximity sensors forming a grid of sensors underneath the keys, theproximity sensors being configured to measure a proximity of an objectto the respective proximity sensors, and produce a signal representativeof the measured proximity; and a second processor configured to receivethe signal representative of the measured proximity, determine at leastone of a position or movement of the object relative to the proximitysensors based on the respective signal, and produce a signalrepresentative of the determined at least one of position or movement,the signal received from the keyboard including the signalrepresentative of the determined at least one of position or movement;and determining one or more commands or instructions for directingperformance of one or more operations of the apparatus or softwareoperating on the apparatus, the one or more commands or instructionsbeing determined as a function of the signal received from the keyboard.7. The apparatus of claim 6, wherein determining one or more commands orinstructions includes determining one or more commands or instructionsthat effectuate a change in a graphical output presented by a displayduring operation of the software operating on the apparatus.
 8. Theapparatus of claim 6, wherein receiving a signal from a keyboardcomprises receiving a signal from a keyboard in at least two instances,the signal in the first instance comprising a trigger signal, and thesignal in the second instance comprising a gesture signal, wherein inthe first instance, the trigger signal is received in response to a userperforming a predefined trigger gesture movement with respect to thekeyboard, wherein the processor is configured to perform or cause theapparatus to perform, in the first instance, identifying the signalreceived from the keyboard as a trigger signal to thereby turn ongesture recognition and interpretation, and wherein determining one ormore commands or instructions occurs in the second instance afterinterpreting the trigger signal in the first instance, the processorotherwise being configured to ignore the gesture signal.
 9. Theapparatus of claim 8, wherein the object comprises a hand or one or morefingers of a hand of the user, and wherein in the first instanceidentifying the signal as a trigger signal comprises identifying thesignal as being representative of a determined at least one of positionor movement of the hand or one or more fingers corresponding to thepredefined trigger gesture movement of the hand or one or more fingers.10. The apparatus of claim 9, wherein the predefined trigger gesturemovement comprises sliding the one or more fingers from a corner of aback edge of the keyboard along the back edge so that the one or morefingers cover a distance within a predefined range of distances alongthe back edge.
 11. The apparatus of claim 9, wherein the predefinedtrigger gesture movement comprises resting the hand or one or morefingers over a side edge of the keyboard, with the hand or one or morefingers covering all or only half of the side edge of the keyboard. 12.The apparatus of claim 11, wherein the predefined trigger gesturemovement comprises resting the hand or one or more fingers over the sideedge of the keyboard, with a thumb of the hand resting over a front edgeof the keyboard.
 13. A non-transitory computer-readable storage mediumhaving computer-readable program code portions stored therein, thecomputer-readable program portions being configured to direct aprocessor to at least perform or cause an apparatus to at least performthe following: receiving a signal from a keyboard, the keyboardcomprising: an arrangement of keys; a plurality of proximity sensorsforming a grid of sensors underneath the keys, the proximity sensorsbeing configured to measure a proximity of an object to the respectiveproximity sensors, and produce a signal representative of the measuredproximity; and a processor configured to receive the signalrepresentative of the measured proximity, determine at least one of aposition or movement of the object relative to the proximity sensorsbased on the respective signal, and produce a signal representative ofthe determined at least one of position or movement, the signal receivedfrom the keyboard including the signal representative of the determinedat least one of position or movement; and determining one or morecommands or instructions for directing performance of one or moreoperations of the apparatus or software operating on the apparatus, theone or more commands or instructions being determined as a function ofthe signal received from the keyboard.
 14. The computer-readable storagemedium of claim 13, wherein determining one or more commands orinstructions includes determining one or more commands or instructionsthat effectuate a change in a graphical output presented by a displayduring operation of the software operating on the apparatus.
 15. Thecomputer-readable storage medium of claim 13, wherein receiving a signalfrom a keyboard comprises receiving a signal from a keyboard in at leasttwo instances, the signal in the first instance comprising a triggersignal, and the signal in the second instance comprising a gesturesignal, wherein in the first instance, the trigger signal is received inresponse to a user performing a predefined trigger gesture movement withrespect to the keyboard, wherein the computer-readable program portionsare further configured to direct the processor to perform or cause theapparatus to perform, in the first instance, identifying the signalreceived from the keyboard as a trigger signal to thereby turn ongesture recognition and interpretation, and wherein determining one ormore commands or instructions occurs in the second instance afterinterpreting the trigger signal in the first instance, the processorotherwise being configured to ignore the gesture signal.
 16. Thecomputer-readable storage medium of claim 15, wherein the objectcomprises a hand or one or more fingers of a hand of the user, andwherein in the first instance identifying the signal as a trigger signalcomprises identifying the signal as being representative of a determinedat least one of position or movement of the hand or one or more fingerscorresponding to the predefined trigger gesture movement of the hand orone or more fingers.
 17. The computer-readable storage medium of claim16, wherein the predefined trigger gesture movement comprises slidingthe one or more fingers from a corner of a back edge of the keyboardalong the back edge so that the one or more fingers cover a distancewithin a predefined range of distances along the back edge.
 18. Thecomputer-readable storage medium of claim 16, wherein the predefinedtrigger gesture movement comprises resting the hand or one or morefingers over a side edge of the keyboard, with the hand or one or morefingers covering all or only half of the side edge of the keyboard. 19.The computer-readable storage medium of claim 18, wherein the predefinedtrigger gesture movement comprises resting the hand or one or morefingers over the side edge of the keyboard, with a thumb of the handresting over a front edge of the keyboard.